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Abstract 

Global competitiveness of the United States is often suggested as 
a key outcome of developing a capable science, technology, engi¬ 
neering, and mathematics (STEM) workforce, a goal supported 
by many local, state, and national programs. Examining the 
effectiveness of such programs, however, may require assessment 
techniques that are outside their organizers’ expertise. The physi¬ 
cists conducting the physics outreach program in the current 
study partnered with education researchers at the same univer¬ 
sity to achieve a more thorough measure of program effective¬ 
ness while also demonstrating how such partnerships represent 
an opportunity to add rigor to current evaluation. The resulting 
analyses demonstrated that participants in the outreach program 
(a) were more likely than nonparticipants to pursue an education 
and career in STEM, (b) were able to define and execute plans to 
solidify a strong foundation for pursuing a career in STEM, and 
(c) persisted in pursuing education in STEM after high school 
graduation. 

Introduction 

£ xposure to different types of careers can support students 
in developing a vision for their future and understanding 
the steps and choices necessary to achieve those careers 
(Hamilton & Hamilton, 2006; National Center for Education Statistics, 
2011). However, many students may not successfully transition into 
their envisioned careers, especially professions in mathematics and 
science, despite being academically prepared. High school students 
who are unclear about their occupational futures may find that they 
over- or under-estimate the amount of education they will need for 
the type of work they wish to pursue and lack a strategic plan for 
accomplishing their goals (Csikszentmihalyi & Schneider, 2000; Sabates, 
Harris, & Staff, 2011; Schneider & Stevenson, 1999). Students may also 
lack role models and experiences that impart the knowledge and 
strategies needed to transition from high school to postsecondary 
education and to realize their career ambitions (Rosenbaum, 2001). 
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The years in which adolescents transition from high school 
to postsecondary education are particularly critical for developing 
ambitions and academically preparing for educational and occu¬ 
pational futures in the fields of science, technology, engineering, 
and mathematics (STEM). Several key policy reports in the United 
States have called for significant steps to increase the national 
STEM labor force (National Academies, 2010; National Science Board, 
2010), and several initiatives and reforms are being implemented 
from preschool to college to encourage and inform students about 
opportunities in STEM. The National Science Foundation (NSF), a 
significant source of funding for strengthening the STEM pipeline, 
spent over $1 billion in 2010 on education funding (Gonzalez, 2012). 
Options for addressing STEM labor shortages are complex, but it 
is clear that STEM experiences should be integrated throughout 
the educational system (National Research Council, 2011). Although 
there is and should be significant attention given to closing gaps in 
the pipeline for those students traditionally underrepresented in 
STEM, there is also significant value in encouraging already inter¬ 
ested and motivated students to realize their ambitions of pursuing 
a STEM career. This study evaluates the role of precollege outreach 
programs in the support and development of talent in STEM fields. 

Uniquely positioned as a bridge between universities and stu¬ 
dents in elementary and secondary schools, precollege outreach 
programs can play an important part in meeting the growing 
demand for an increased labor supply in STEM. The goals of the 
present study were twofold. The primary objective was to examine 
one college-outreach program dedicated to STEM outreach, the 
Physics of Atomic Nuclei (PAN), which seeks to expose students 
to careers in nuclear science and offer them practical research 
experience. The secondary goal was to examine how the interdis¬ 
ciplinary partnership between an outreach program in science and 
researchers in social science can improve the thoroughness of the 
program’s overall evaluation in regard to its standing and effective¬ 
ness while also contributing to the research base as it relates to the 
role of outreach programs. 

The secondary objective was motivated by research that has 
shown a lack of STEM precollege programs that are able to carry 
out statistically appropriate evaluations. Such programs often 
struggle to demonstrate their ultimate impacts (U.S. Department of 
Education, 2007). Although the present study does not intend to pro¬ 
vide an exhaustive review of collaborative evaluations conducted 
on precollege programs, a brief search conducted using Google 
Scholar with the keyword phrases “pre-college,” “evaluation,” and 
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“STEM” returned a mixture of different types of research studies 
appearing since 2008. The predominant mode of evaluating precol¬ 
lege outreach programs is the analysis of self-reported survey mea¬ 
sures through either cross-sectional data collection or a pre-post 
design. These data can inform several areas in the evaluation of 
the outreach programs’ effectiveness, as many of these studies have 
suggested; however, such studies also have limitations, such as the 
subjectivity of participants. Additionally, current research designs 
have often not been able to accurately capture any knowledge or 
skill acquisition that reflects outreach participants’ experiences 
in the program. As a result, several studies also incorporated the 
use of an assessment to strengthen their program evaluation (e.g., 
Bogue, Shanahan, Marra, & Cady, 2012). A third mode of evaluation 
used to understand the impact of precollege outreach in STEM 
was the examination of undergraduate students who reported their 
participation in a precollege outreach program prior to college (e.g., 
Cohen & Deterding, 2009). These different modes of inquiry provide 
several perspectives into how students benefit from particular out¬ 
reach programs. Ascertaining how a student would have responded 
without the treatment might theoretically be achieved by ran¬ 
domizing students to participate in these programs; such an act, 
however, could violate the very goals of the outreach. The present 
study used statistical methods to create a comparison group to 
examine how this program impacted its participants compared to 
nonparticipants. 

To evaluate the influence of PAN on student participants, this 
study investigated three research questions: (1) What are the ben¬ 
efits of participation reported by the students?; (2) How do the 
PAN participants compare to a statistically matched sample from 
national data; specifically, what are the differences in educational 
and occupational interests conditioning on student characteris¬ 
tics?; and (3) What are the long-term implications for participants 
as reflected in follow-up data? Our multiple analyses offered evi¬ 
dence that students who participated in PAN were not only more 
likely to be interested in majoring in STEM than members of their 
matched comparison group, but that PAN gave them the knowl¬ 
edge, skills, and strategies necessary to successfully plan and pursue 
careers in physics and other STEM fields. 

Connecting Talented Students With Potential 

Careers 

To support national competitiveness in an increasingly tech¬ 
nology-driven global economy, the U.S. government has invested 
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heavily in developing a STEM workforce to meet future demand. 
Students interested in pursuing careers in growing fields can meet 
this need. As adolescents plan their future education and occupa¬ 
tions, knowledge of different careers can help them identify and 
take pragmatic steps to achieve their career goals ( Rosenbaum, 2001; 
Schneider & Stevenson, 1999). However, this process of developing 
interests and pursuing the steps necessary to achieve career goals 
is not always carried out. 

Of the students who finish high school academically prepared 
to pursue a variety of career opportunities, many never reach their 
occupational goals. Such students may falter because they lack 
access to the necessary information for developing a strategic plan 
to achieve their goals and may under- or overestimate the level 
of education they need (Schneider & Stevenson, 1999). The ability of 
students to attain their career goals can be impeded if they are not 
able to articulate and understand the appropriate pathways from 
secondary education through college and into a chosen career. This 
alignment of ambitions and the education required to attain them 
is especially important for students pursuing careers in STEM; in 
prior research, students have reported difficulty in identifying real¬ 
istic strategies that would help them achieve their career goals, such 
as how to select coursework and extracurricular activities appro¬ 
priate for STEM careers (Csikszentmihalyi & Schneider, 2000; Schneider 
& Stevenson, 1999). Parents and family are also important factors 
in the way students develop future goals and ambitions (Hossler 
& Stage, 1992). Like these students, parents and family members 
may lack information about the specific requirements, options, or 
courses needed for acceptance by more selective colleges or the 
preparation needed for specific STEM careers. 

University-based outreach programs can supplement K-12 
education in improving several student outcomes, especially in 
STEM, including interest, knowledge, skills, and the development 
of postsecondary expectations (National Academies, 2007). Some 
programs provide STEM-related professional development for 
teachers, leading to more effective instruction for students (Moskal 
& Skokan, 2011). Outreach through summer camps in particular 
can be a successful strategy to advance student outcomes in STEM 
(Foster & Shiel-Rolle, 2011). However, there is a dearth of studies that 
demonstrate their effectiveness in improving attitudes about and 
understanding of scientific pursuits. 

Prior studies that have followed students from adolescence into 
adulthood showed that the selection of postsecondary institution, 
choice of college major, and interest in pursuing a career in STEM 
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were associated with being able to visualize oneself as a college stu¬ 
dent, transform interests into realistic actions, and create strategic 
plans (Schneider, Judy, & Mazuca, 2012). The current study used this 
framework of visualization, realistic actions, and strategic plans to 
understand how college-outreach programs can support each step 
in this transition process by (a) providing opportunities for stu¬ 
dents to visualize themselves as college students, often through on- 
campus experiences; (b) suggesting realistic actions and activities 
for students to hone their interests; and (c) supporting the devel¬ 
opment of strategic plans so students can successfully attain their 
educational and career goals. 

Physics of Atomic Nuclei Program (PAN) 

PAN is a precollege outreach program that informs students 
about careers in nuclear science and offers authentic research 
experiences for student participants. This program is a partner¬ 
ship between two NSF-funded entities: the Joint Institute for 
Nuclear Astrophysics (JINA) and the National Superconducting 
Cyclotron Laboratory (NSCL). JINA is a multi-institutional NSF 
Physics Frontier Center, a large collaboration between the primary 
partners of three Midwestern universities and other national and 
international partner organizations. JINA brings together nuclear 
experimentalists, nuclear theorists, and astronomers to collaborate 
on problems related to nuclear astrophysics. PAN is fully funded 
by JINA as one of its premier outreach programs, making it free to 
participants. 

PAN is hosted by a world-leading rare isotope research facility 
located on the campus of a large Midwestern university. This nuclear 
laboratory hosts over 1,300 users from 35 countries and maintains 
one of the top-ranked nuclear science graduate programs. The lab 
contributes facilities; equipment; and additional faculty, staff, and 
students to support PAN. Hundreds of students from across the 
United States and some foreign countries have been attracted to 
PAN over its 20-year history. Throughout the program’s evolution, 
the primary goal has always been to build student interest and 
knowledge in the field of nuclear physics. Student participants are 
prospective future researchers and are treated as such. 

Many precollege outreach programs focus on exposing stu¬ 
dents from underrepresented groups to STEM careers. PAN seeks 
and accepts students from such groups, but the nature of the pro¬ 
gram draws applicants who typically are highly intelligent, science- 
focused, and self-motivated. It is recognized that these students 
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usually have significant resources already, such as supportive and 
knowledgeable parents and access to advanced learning opportu¬ 
nities. Rather than introducing participants to science in general, 
PAN aims to support and channel their enthusiasm for science into 
potential college majors and specific careers in research. 

PAN consists of an intensive week-long experience in the 
summer, featuring days packed with up to 12 hours of program¬ 
ming. Daily activities include one or two introductory-level lec¬ 
tures by JINA/NSCL faculty regarding current research, experi¬ 
mental methods, and detector technology. Each lecture is followed 
by a structured question and answer session where teams of par¬ 
ticipants can interact with the presenter. Through the students’ 
evaluation of what was clear and unclear in the presentation, the 
presenter obtains valuable feedback, and the students gain further 
clarifications. Before beginning any research in the lab, training 
sessions are held to ensure that participants can safely and effec¬ 
tively pursue the research program. Students then conduct a series 
of experiments using a $1 million scintillation detector. They 
learn to use this detector to identify the location of a radioactive 
source, gain experience with the data-acquisition software, and 
track cosmic rays as they pass through the detector. Through this 
sequence of steps, they build their knowledge and confidence until 
they begin pursuing research in much the same way NSCL users 
would, making it an authentic experience. Finally, students design 
and present a poster explaining their research at the capstone ses¬ 
sion of the program. This exercise teaches the student participants 
what content is appropriate for reporting results and familiarizes 
them with methods of presentation, both of which are fundamental 
and integral components of research at any level. Evening programs 
supplement the above activities by acclimating the students to 
campus life. These auxiliary programmatic events help the partici¬ 
pants visualize themselves as college students in this field, engage 
in college-level work, and continue their development of strategic 
plans for postsecondary pathways and opportunities in physics. 

Ultimately, PAN is intended to help students see themselves 
pursuing a research career by (a) promoting the importance of 
nuclear research; (b) teaching the discipline and current topics of 
nuclear astrophysics; (c) introducing students to undergraduate/ 
graduate life at a university; (d) demonstrating the nature of pure 
science and research careers, particularly in nuclear science; and 
(e) fostering interest in nuclear physics/astrophysics. 
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Because JINA and NSCL receive grants from the NSF, pre¬ 
college outreach programs like PAN are intended to fulfill NSF’s 
Broader Impacts goals in these ways: 

1. promote teaching, training, and learning by intro¬ 
ducing nuclear astrophysics to new audiences; 

2. broaden participation among underrepresented 
groups by recruiting more female students for the 
program; 

3. enhance education partnerships by providing teachers 
and students with long-term support: assistance, mate¬ 
rials, and further outreach opportunities; 

4. disseminate information broadly by sharing it with the 
teachers of tomorrow’s scientists; and 

5. benefit society by encouraging science literacy and 
knowledge of how nuclear astrophysics affects society. 

PAN sought and earned approval from the university’s Institutional 
Review Board to collect student survey data and use it for improve¬ 
ment and research. Following all assent/consent and confidenti¬ 
ality guidelines, PAN has amassed many years of data about the 
effectiveness of the program through exit surveys and anecdotal 
evidence from alumni, but lacked expertise in assessing the success 
of the program relative to its goals. Some efforts to this effect have 
been made in recent years in collaboration with other precollege 
programs on campus. However, the program director and advisory 
committee recognized a need for a more detailed analysis to satisfy 
the NSF mandate, and thus formed a partnership with the College 
of Education at their university. 

Over the past 3 years of this partnership, the survey instru¬ 
ments have been updated to not only include students’ self-reports 
of their experiences, but also new items drawn from national sur¬ 
veys that have been validated and will allow PAN participants 
to be compared to a control group—a necessary comparison for 
understanding the impacts of the outreach and for demonstrating 
evidence of its effectiveness. 


Method 


Sample 

Because PAN intends to inspire students to study science at the 
college level, and particularly to give them experience in research 
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so they can make an informed decision about their future careers, 
the preferred candidates are those who show a clear desire and 
drive to pursue science. Students demonstrate this on their appli¬ 
cation by indicating such academic and extracurricular choices 
in high school as regularly enrolling in science and math courses, 
joining science-related clubs and similar extracurricular activities, 
committing to science programs and camps outside school, seeking 
out reading and other material to further their science knowledge, 
and applying to PAN as a way to investigate a science career. It 
is important to note that the application requires no GPA or test 
scores. 

Applications must also include two teacher recommenda¬ 
tions, which are rated by the following criteria: whether the teacher 
instructed the student in science/math; amount of interaction 
between the teacher and student; specific instances where the stu¬ 
dent has “gone the extra mile” to learn more; the teachers rating 
of how inquisitive the student is, particularly if the teacher cannot 
answer all of his/her questions; and the teacher’s rating of the stu¬ 
dent’s motivation, independence, and maturity. 

As one of the few summer programs in nuclear science, PAN 
is quite popular for students who are seeking that specific experi¬ 
ence. Due to budget and equipment limitations, the program can 
accept only 24 students per year. To help promote gender parity, 
an equal number of male and female participants are selected. It is 
generally (and usually correctly) assumed that students seeking out 
PAN have a keen interest in science but little to no background in 
nuclear astrophysics, and thus all students can begin their instruc¬ 
tion from the same level of knowledge and be successful. 

Table 1 shows the applicant pool from 2007-2012. Over those 
6 years, the applicant pool increased in size, and the acceptance 
rate declined as the program became necessarily more selective. In 
2012,179 students applied for the 24 spots, leading to an acceptance 
rate of just 13%. Overall, consistent patterns of applicants can be 
observed from year to year: more males than females apply, there 
is generally a larger proportion of in-state students compared to 
out-of-state applicants, and White students make up the majority 
of the pool. Given the sharp increase in applications in later years, 
it is important to note that between the 2008 and 2009 programs, 
PAN organizers completely changed advertising strategies. Before 
2009, PAN was promoted by physically mailing letters to in-state 
high schools, but in early 2009, it was featured in several teaching 
and physics magazines as well as through electronic media. This 
campaign reached a wider audience, and eventually PAN transi- 
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tioned from having only an in-state presence to gaining a broader 
national audience. This increased exposure created larger appli¬ 
cant pools and resulted in a more demanding selection process so 
that the admitted students tended to be highly self-motivated with 
extreme interest in science. 


Table I. Demographic Characteristics Applicants 





Percentage of Applicants 

Applications 

by Year 

Total 

Applications 

Acceptance 

Rate 

Gender 

Race/Ethnicity* 

Location 




M 

F 

W 

B 

H 

A 

M 

O-O-S 

2007 

47 

51.06% 

62% 

38% 





87% 

13% 

2008 

37 

64.86% 

73% 

27% 

- 

- 



86% 

14% 

2009 

91 

26.38% 

63% 

37% 

70% 

5% 

3% 

20% 

85% 

15% 

2010 

82 

20.27% 

60% 

40% 

72% 

7% 

13% 

22% 

55% 

45% 

2011 

127 

18.90% 

76% 

24% 

70% 

8% 

6% 

17% 

59% 

41% 

2012 

179 

13.41% 

76% 

24% 

68% 

8% 

5% 

17% 

49% 

51% 


Note. Race/ethnicity data not collected in 2007 and 2008. M= Male, F= Female, 
W= White, B= Black, H= Hispanic,A= Asian, M= Michigan, 0-0-S= out-of-state. 


To allow for comparative analyses of PAN and these rigorously 
selected students, a comparison sample of students from a national 
data set, the Education Longitudinal Study: 2002 (ELS:2002) was 
used. This data set is a nationally representative longitudinal study 
designed to help understand the transition of adolescents from 
high school through postsecondary school and work. This data set 
represents the most recent available longitudinal study with post¬ 
secondary outcomes. Currently, three waves of data are available: 
the base year sample from 2002, when students were in the 10th 
grade; the first follow-up during their 12th grade year in 2004; and 
a second follow-up in 2006. In the base year, over 16,000 students 
were surveyed in over 750 schools. A restricted sample of students 
from this data set was used as a comparison control group; the 
selection of this sample is detailed in the following section. 

Data and Analysis 

Data in this study were drawn from a diverse set of sources, 
including student surveys, qualitative participant data, follow-up 
data from former participants (“PAN alumni”), and, for the com¬ 
parison group, data from ELS:2002. To address the first research 
question of how students benefit from participation in PAN, mul¬ 
tiple years of survey data were descriptively analyzed. Data col¬ 
lected on student survey instruments included (a) satisfaction 
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with PAN activities, (b) perceived benefits of their participation, 
(c) postsecondary and career aspirations, and (d) open-response 
items regarding the influence of PAN on their educational and 
occupational plans. To observe the long-term implications for 
PAN participants, follow-up data obtained from an online survey 
of PAN alumni was descriptively analyzed to ascertain whether 
participants continued in their pursuit of physics or other STEM- 
related majors in college and their current occupational goals. 

Because of the select nature of PAN participants, to address 
the second research question of the program’s effectiveness on the 
outcomes of educational and occupational interests in STEM, a 
comparison group of students was created from ELS:2002 using 
propensity-score matching techniques. Participants with complete 
survey responses who participated in the PAN programs in 2011 
and 2012 (n = 30) were matched to students from ELS:2002 data 
on a comprehensive set of variables that included gender, race, atti¬ 
tudes toward math and reading, parental education level, parent 
interaction characteristics, and postsecondary ambitions and 
behavior. The selectiveness of the PAN acceptance criteria created 
a very narrow group of students in the ELS:2002 sample that sta¬ 
tistically met the criteria, such that a group of 38 students were 
identified as the control group using nearest neighbor matching 
and after covariate balance was achieved. 

Results 

Table 2 shows that not only did students enjoy the program, 
but over time, there was an increase in the percentage of students 
reporting that PAN influenced their interest in science, educational 
goals, and career plans. Participants had already self-identified as 
having an interest in science, so it is not surprising that they dem¬ 
onstrated high satisfaction with their research experience at PAN. 
Indeed, it proved difficult to measure changing attitudes toward 
science given that PAN participants already had greater interest in 
science than the average high school student. 

PAN staff used survey feedback on specific PAN activities to 
improve programmatic components over the years and thus pro¬ 
vide a better experience for students. Common trends from survey 
results drove two very important changes to PAN. First, partici¬ 
pants highly valued the parts of the program that included contact 
with faculty and graduate students. PAN activities have therefore 
evolved to maximize opportunities for participant interaction with 
faculty and graduate students. The second highly valued aspect of 
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PAN was having an increasingly authentic research experience. 
Thus, new training sessions and experiments with the scintillation 
detector were introduced. The increased opportunities to work 
with faculty and graduate students in an environment providing 
authentic, hands-on research experiences help students visualize 
themselves as scientists and researchers. The visualization com¬ 
ponent is crucial to their experience, as it can illuminate areas of 
their own knowledge and skill development that they might need 
to improve through strategic planning while in high school, such 
as how they select their coursework. 


Table 2. Descriptive Analysis of Participants 



Percentage of students who agreed that PAN... 








Total 

Increased interest in science 








Influenced career plans or 
future course selections 

68% 

70% 

96% 

96% 

100% 

96% 

88% 

Provided overall enjoyment 
of the program 

89% 

96% 

92% 

100% 

100% 

100% 

96% 


Source: PAN Student Survey 


Over the years, it has been common for students to report 
that their perception of research careers underwent a dramatic 
shift during PAN. Experiencing the life of a researcher for one¬ 
self proved to be a powerful catalyst, especially when there was a 
large difference between perception and reality. Being placed in 
the role of a researcher and developing relationships with current 
researchers allowed PAN participants to identify with a previously 
unknown or misperceived career. The experience thus supported 
the transformation of a participant’s general interest in science to a 
more specific and active one, wherein students began to visualize 
themselves in this career and were able to take strategic steps to 
become future scientists based on their personal experiences in the 
field. Open-ended responses from the PAN participant and alumni 
surveys were coded into two themes consistent with the visualiza¬ 
tion, strategic planning, and realistic actions framework: (a) their 
perceptions about careers in physics/STEM and their visualization 
of themselves as a future researchers and (b) the development of 
their strategic plans to pursue a career in physics/STEM. 

One participant noted that “PAN made me realize how inno¬ 
vative scientists need to be, and to think outside the box.” A PAN 
alumnus refuted a common stereotype of a scientist working alone 
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and in isolation: “My favorite part of PAN was the fact that it incor¬ 
porated teamwork, lecture, and hands-on research; this allowed us 
to work all parts of our brain as it simulated a real world multifac¬ 
eted research experience.” A crucial aspect of PAN is taking stu¬ 
dents who have an interest in science or physics and helping them 
develop that general interest into a career. One alumnus reflected 
on the influence of the program: 

I think the PAN program was great! I knew I had a 
peripheral interest in science as a high school stu¬ 
dent, but PAN showed me that this interest was worth 
pursuing in college. Now I’m a graduate student in 
astronomy, and I think getting involved in science early 
on had a great deal to do with my career choice now. 

Again, PAN students are already highly interested and motivated 
students, but PAN supports the sharpening of their focus in regard 
to defining next steps and creating strategic plans so that they can 
solidify a strong foundation for pursuing a career in STEM, par¬ 
ticularly in nuclear astrophysics. 

There were, however, a few alumni who noted that PAN led 
to the discovery that experimental physics was not for them. Such 
an assessment still helps students develop future educational and 
occupational plans in the positive sense, leaving open the possi¬ 
bility of work in a different area of science. 

Table 3 provides a summary of the PAN alumni postsecondary 
interests. All of the former participants report planning to attend, 
attending, or graduating from college. Of the alumni who completed 
the follow-up survey, 87% planned to major in or were currently 
majoring in STEM, 40% planned to major in or were currently 
majoring in physics, and 40% had the educational expectation 
of pursuing a Ph.D. Although only 152 former participants were 
able to be contacted for follow-up, and not all of those contacted 
responded, these percentages still provide evidence as to the oppor¬ 
tunities that participants pursue after high school. 

Table 3. PAN Alumni Outcomes 


n = 45 respondents 


Planning to attend, attending, or graduated from a 4-year college 


Planning to major in or majoring in STEM 

87 

Planning to major in or majoring in physics or astrophysics 

40 

Planning to pursue a Ph.D. 

40 


Note. Data from the PAN Alumni Follow-Up Survey 2012. 
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Due to the highly selective nature of the PAN program, selec¬ 
tion bias made estimating effects of the program challenging. 
Propensity score methods can provide one way to compensate 
for this bias. The primary advantage of using propensity score 
matching is that the likelihood of participating in PAN for nonpar¬ 
ticipants (or students with similar potential to attend PAN) can be 
estimated given the characteristics of the PAN students (Schneider, 
Carnoy, Kilpatrick, Schmidt, & Shavelson, 2007). This essentially cre¬ 
ates statistically equivalent groups of students that have equal pro¬ 
pensities to be in the treatment group. Table 4 shows the similari¬ 
ties between the PAN students (treatment) and ELS:2002 students 
(control). 


Table 4. PAN Participants and ELS.-2002 Comparison Group 



Students 

Significance Test 





p-value 

Total number of students (n) 

Student Background Characteristics (%) 





Male 

50% 

45% 

0.19 

0.67 

Minority (Black and Hispanic) 

10% 

8% 

0.09 

0.76 

Student Educational Expectations (%) 





Complete a master’s degree 

13% 

30% 

2.56 

0.1 1 

Complete a Ph.D. 

87% 

70% 



Student Attitudes and Behaviors (%) 





Gets totally absorbed in mathematics 

100% 

95% 

3.38 

0.18 

Thinks math is fun 

97% 

97% 

0.04 

0.98 

Thinks math is important 

97% 

97% 

1.23 

0.54 

Born with math ability 

43% 

43% 

1.57 

0.67 

Gets totally absorbed in reading 

97% 

97% 

0.67 

0.71 

Thinks reading is fun 

97% 

97% 

1.45 

0.48 

Reads in spare time 

93% 

89% 

4.28 

1.45 

Parent Characteristics (%) 





At least 1 parent holds a bachelor’s degree 

87% 

89% 

0.13 

0.72 

Parent frequently checks homework 

0% 

0% 

* 

* 

Student frequently discusses grades with 

77% 

89% 

2.03 

0.15 

parent 





Student frequently discusses courses with 

67% 

87% 

2.03 

0.15 

parent 
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Students 

Significance Test 





p-value 

Student frequently discusses college wtih 
parent 

97% 

97% 

0.03 

0.87 

Average number of postsecondary 
institutions applied to 

8 

7 

-2.02** 

0.05 


Note. Data from NCES Education Longitudinal Study of 2002 (ELS.2002) and PAN Student 
Survey 2001 & 2001. 

* No test performed, exactly equal. ** t-statistic from a two-sample test (equal 
variances). 


On almost all covariates, there were no significant differences 
between the groups. However, two variables did show significant 
differences between groups. One was the frequency of students’ 
discussing courses with their parents. PAN students showed 
slightly less frequency for this behavior, which may be due to dif¬ 
ferences in high schools between 2002 and 2012. Particularly in 
the state where this program is located, credits and standards have 
increased, so students have fewer choices regarding their course¬ 
taking. Currently, students must take 3 years of science and 4 years 
of math, including Algebra II. These more standardized coursework 
options might mean students have less reason to discuss course 
decisions with parents. The second variable that was statistically 
different was the number of postsecondary institutions to which 
students applied. We argue that this is not practically significant— 
a student who applies to seven institutions looks largely similar to 
one who applies to eight (with equal variance). 

Given the statistically insignificant differences between these 
two groups, the final analysis used the treatment variable PAN to 
estimate the effect of participation using a logistic regression on 
the two outcomes: interest in STEM major (1 = yes and 0 = no) 
and STEM career (1 = yes and 0 = no). As reported in Table 5, 
PAN students’ likelihood of pursuing a major in STEM was almost 
nine times that of nonparticipants. For students who participated 
in PAN, the likelihood of desiring a career in STEM was eight times 
that of the control group of students. All models were estimated 
with robust standard errors, and student covariates were not used 
as explanatory variables for either model because these controls 
were used to create the groups. Sensitivity analyses using these 
background characteristics in the models only increased the effect 
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of PAN; thus, the ratios presented represent the most conservative 
estimates. 


Table 5. Logistic Regression of PAN Participants on STEM Major and 
Career 



Major in STEM 

STEM Career 


Odds Ratio SE p-value 

Odds Ratio SE p-value 

PAN Program 




Note. Data from NCES Educational Longitudinal Study of 2002 (ELS:2002) & PAN Student 
Survey 201 I & 2012. 


Discussion 

Evaluation of the PAN program is key for identifying strengths 
and weaknesses in implementation to improve programmatic fea¬ 
tures each year. Furthermore, such evaluation is critical to the 
funding and sustainability of outreach programs similarly aiming 
to strengthen the future opportunities of individuals in the STEM 
fields. PAN is an example of how outreach programs can specifi¬ 
cally respond to the call to strengthen the pipeline of talent into 
STEM by helping students visualize, take realistic actions, and 
create strategic plans to pursue a career in physics and STEM. As 
described in this study, PAN provides students who are already 
highly interested in science with an experience that can solidify 
their interest into pursuing majors and careers in relevant fields. 
Similar students who are interested in physics and STEM who have 
not had such opportunities to gain insight into the field or hands- 
on experience in this career might fail to take the necessary steps 
to align their ambitions with postsecondary and occupational goals 
and may “leak” from this STEM pipeline. 

This study presents possible evidence of the short-term and 
long-term impact of PAN on participants. In the short term, partic¬ 
ipants reported high levels of satisfaction with PAN and provided 
evidentiary support for how PAN encouraged the development and 
realization of their postsecondary and career goals. Comparing 
PAN students to similar students who did not participate in PAN 
illustrated the strong influence of PAN on the outcomes of pur¬ 
suing a STEM major and a career in the STEM field. In the long 
term, a significant majority of PAN participants remain committed 
to pursuing a future in STEM. Although some alumni reported 
that PAN helped them decide not to pursue a career in physics or 
STEM, the program nonetheless supported the students’ develop- 
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ment of educational and occupational plans, though potentially in 
a different field of study. 

It is worth noting that our findings are suggestive rather than 
conclusive about the effects of PAN. Although we argue that the 
statistical approaches used provide strong inferential evidence of 
the impact of PAN for participants, this study has several limita¬ 
tions. First, we acknowledge the selective nature of the application 
process and recognize that when selecting our comparison sample, 
we were unable to control for both the interest in physics specifi¬ 
cally and the application process of attending PAN. This study was 
also limited by the small sample size of 24 participants each year. 
Although we were able to combine the previous 2 years of partici¬ 
pant data, we also lost participants from the sample who did not 
complete all of the necessary pre- and postsurvey questionnaires. 
However, we argue that since these students are largely similar in 
background characteristics and interests, this response rate does 
not alter the overall analysis. Lastly, not all alumni responded to 
the follow-up survey. This shortage of alumni responses could con¬ 
tribute to an overrepresentation of PAN alumni in STEM because 
the alumni who continued to pursue physics or other related 
fields may be more likely to respond than those who are no longer 
interested. 

Although the indication that PAN plays a role in students’ 
selection of STEM careers proves satisfying for the program orga¬ 
nizers, both the process of assessment and its ultimate result will 
have profound impacts on the future of PAN and other similar 
programs. As stated earlier, the analysis of the data was significantly 
delayed by the inexperience of those who collected it. The identifi¬ 
cation of a control group for comparison with the PAN participants 
proved singularly difficult. The requirements for generating such 
a group and the kinds of comparisons available will influence the 
kinds of survey data that will be collected in subsequent programs. 
Organizers will actively seek assistance from assessment experts 
for subsequent analyses. Other, less established programs would 
benefit from such scrutiny much earlier in their tenure. The STEM 
precollege landscape is not suffering from a dearth of programs, 
but it may find extraordinary value and the affirmation of funding 
agencies in the pursuit of connections to education researchers and 
a data-driven evaluation. A sister program at a nearby university 
reproduces much of the same experience for their summer out¬ 
reach program and will likely take advantage of lessons learned 
through this study. 
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PAN exit survey results have consistently shown that it is 
impossible to satisfy everyone. Over the years, many participants 
have questioned program choices and have made thoughtful sug¬ 
gestions. PAN organizers have always debated whether requested 
changes would truly maximize program impact or simply make 
the participants happier. The principles of faculty interaction and 
laboratory immersion will continue to drive decision-making for 
future changes as PAN evolves. The consistent support for these 
two aspects in survey responses and the way they have been cited 
by alumni demonstrate their effectiveness and will make it easier 
to sift through options for PAN in the coming years. Indeed, the 
process of assessing PAN has brought into sharper focus (a) who 
the target audience is and (b) the PAN strategy for reaching that 
audience. That this was made possible by a cross-campus partner¬ 
ship also suggests that arenas for many similarly productive col¬ 
laborations are just waiting to be discovered. 
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